Skip to content

PERF: Quick Shift Implementation (GH5609) #6672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 20, 2014
Merged

PERF: Quick Shift Implementation (GH5609) #6672

merged 1 commit into from
Mar 20, 2014

Conversation

gouthambs
Copy link
Contributor

closes #5609

Test code

import pandas as pd
import numpy as np
import timeit

xdim = 10000
ydim = 500
repeats = 10
df = pd.DataFrame(np.random.rand(xdim,ydim))

# axis 0 shift
s = timeit.default_timer()
for i in range(repeats):
    df1 = df.shift(1,axis=0)
e = timeit.default_timer()
print "Axis 0 shift",e-s

# axis 1 shift
s = timeit.default_timer()
for i in range(repeats):
    df1 = df.shift(1,axis=1)
e = timeit.default_timer()
print "Axis 1 shift",e-s

Results

V0.13.1

Axis 0 shift 0.958936203491
Axis 1 shift: IndexError: index 500 is out of bounds for size 500

----------------------------------------------------------------------------------
V0.13.1-iss5609 branch

Axis 0 shift 0.1622
Axis 1 shift 0.3466

@gouthambs gouthambs mentioned this pull request Mar 20, 2014
Closed
@gouthambs gouthambs changed the title Iss5609 Quick Shift Implementation - Issue 5609 Mar 20, 2014
jreback
jreback reviewed Mar 20, 2014
View reviewed changes
pandas/core/internals.py
@@ -1910,9 +1907,20 @@ def fillna(self, value, limit=None, inplace=False, downcast=None):
values = self.values if inplace else self.values.copy()
return [self.make_block(values.get_values(value), fill_value=value)]

def shift(self, indexer, periods, axis=0):
""" shift the block by periods """
@classmethod
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just make this a function defined inside shift itself. if its truly reusable them move to core/common, but I don't think used anywhere else, so just move to inside.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SparseDataFrame is using this shift_indexer. However I don't think it is reusable enough to move to core/common. I plan to revert back to original code. What do you think?

@jreback
Copy link
Contributor

jreback commented Mar 20, 2014

looks good....

if you feel comfortable squashing go ahead

otherwise you can ping me when its green

@jreback jreback changed the title Quick Shift Implementation - Issue 5609 PERF: Quick Shift Implementation (GH5609) Mar 20, 2014
@gouthambs
A merged version of all my commits .
1c5e779 
Added to note release.rst

Move shift_indexer into the shift function

Removed function _shift_indexer all together. Passes tests sparse.

remove unwanted comments
@gouthambs
Copy link
Contributor Author

I have squashed my commits.

@jreback
Copy link
Contributor

jreback commented Mar 20, 2014

looks good...ping me when green

@gouthambs
Copy link
Contributor Author

Good to go!

@jreback jreback added this to the 0.14.0 milestone Mar 20, 2014
jreback added a commit that referenced this pull request Mar 20, 2014
@jreback
Merge pull request #6672 from gouthambs/iss5609
80cef54 
PERF: Quick Shift Implementation (GH5609)
@jreback jreback merged commit 80cef54 into pandas-dev:master Mar 20, 2014
@jreback
Copy link
Contributor

jreback commented Mar 20, 2014

@gouthambs thanks!

next time will be easier!

@gouthambs gouthambs deleted the iss5609 branch March 20, 2014 20:17
@jreback jreback mentioned this pull request Aug 10, 2014
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Performance Memory or execution speed performance Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Milestone
0.14.0
Development

Successfully merging this pull request may close these issues.

Quick dataframe shift
2 participants
@gouthambs @jreback